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(54) Imaging method and imaging apparatus 

(57) An imaging unit reads an image signal from an 
image pickup device representing a captured Image. A 
signal processor generates image data from the image 
signal and generates audio data by sampling an ana- 
logue audio signal provided with the image signal. A 
controller controls the operation of the imaging untt and 



the signal processor to alter a frame rate of the Image 
data to a predetermined set frame rate and to generate 
associated information for indicating the set frame rate 
or the like. A transmitter combines the associated infor- 
mation with the image data and the audio data to trans- 
mit the combined ones as material data. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The Invention relates to an Imaging method 
and an imaging apparatus. 

Related Art 

[0002] In generation of contents on audio and/or video 
for broadcasting, it has been often done to generate the 
contents altering motion speed of a subject partially for 
only a set period, in order to obtain enhanced effects the 
creator wishes to create. 

[0003] In such the generation of contents altering the 
motion speed of the subject, a higher frame rate relevant 
to a reference frame rate is illustratively set to generate 
the contents, and then the contents are reproduced at 
the reference frame rate so that slow motion contents 
can be produced. Alternatively, a lower frame rate rele- 
vant to a reference frame rate is set to generate the con- 
tents, and then the contents are reproduced at the ref- 
erence frame rate so that quick motion contents can be 
produced. Further, adapting the frame rate to be set and 
the frame rate at the reproduction allows the motion 
speed of the subject to be freely altered. 
[0004] Thus, the creator creates the contents for 
broadcasting using not only the contents generated at 
the reference frame rate but also the contents of altered 
motion speed of the subject, in order to obtain the en- 
hanced effects the creator wishes to create when the 
contents are reproduced at the reference frame rate. 
[0005] It has been proposed in Japanese Patent Pub- 
lication No. H11-177903 that the video camera allows 
time axis to be compressed or extended in order to gen- 
erate the contents of altered motion speed of the su bject 
by altering the frame rate as described above. 
[0006] When the contents for broadcasting are gen- 
erated using not only the contents generated at the ref- 
erence frame rate but also the contents of altered motion 
speed of the subject as material, it is not easily deter- 
mined at editing which of frame rates each of the con- 
tents has been generated if merely the material image 
is displayed. Further, when it performs the speed 
change on the contents of altered motion speed of the 
subject, it is impossible to determine whether or not the 
speed change can be performed without any deteriora- 
tion in image quality of the displayed image. If a frame 
rate is set (hereinafter, referred to as, "set frame rate") 
to 10 times the reference frame rate, the reproduced 
contents are displayed one-tenth in a motion of subject 
if the contents generated at the set frame rate are re- 
produced at the reference frame rate. Here, it is con- 
ceivable that since the set frame rate is 10 times the 
reference frame rate when the creator wants to display 
the contents one-fifth in a motion of subject, the speed 



change can be easily performed without any deteriora- 
tion in image quality of the displayed image if performing 
a frame-skipping for each frame. It, however, is impos- 
sible to determine whether or not the speed change can 
s be performed without any deterioration in image quality 
of the displayed image, easily based on merely the dis- 
played image. 

SUMMARY OF THE INVENTION 

10 

[0007] According to an aspect of the present inven- 
tion, an imaging apparatus comprises imaging device 
for reading a signal captured out of an image pickup de- 
vice so as to generate an Image signal based on cap- 
's tured image, and signal processor for generating image 
data based on the image signal. The apparatus also 
comprises controller for controlling operation of at least 
one of the imaging device and the signal processor to 
set a variable frame rate of the image data to a set frame 
20 rate, and generating associated information for indicat- 
ing at least the set frame rate. The apparatus further 
comprises transmitter for combining the associated in- 
formation with the image data to transmit the combined 
ones. 

25 [0008] Embodiments of the present invention can pro- 
vide an imaging method and an imaging apparatus 
wherein determining which of frame rates the imaging 
has been performed is easily allowed. 
[0009] According to another aspect of the present in- 
30 vention, an Imaging method comprises the steps of gen- 
erating image data based on a signal read out of an im- 
age pickup device, setting a variable frame rate of the 
image data to a set frame rate, generating associated 
information including frame rate information for indicat- 
es ing the set frame rate, and combining the associated in- 
formation with the image data to transmit the combined 
ones. 

[0010] In the present invention, when reading signal 
captured out of an image pickup device so as to gener- 

*o ate image signal based on captured image, and gener- 
ating image data based on the image signal, the con- 
troller controls operation of at least one of the imaging 
device and the signal processor to set a variable frame 
rate of the image data to a set frame rate and generates 

*5 associated information for indicating at least the set 
frame rate. Thus, the transmitter combines the associ- 
ated information with the image data to transmit the 
combined ones. Based on the associated information, 
determining which of frame rates the imaging has been 

50 performed is easily allowed, and thus, the contents of 
altered motion speed of the subject can be reproduced. 
[001 1 ] Further, a variable frame rate of the image data 
is freely set to a set frame rate by means of altering a 
frequency of reading the signal captured out of the im- 
age pickup device, controlling the signal processor to 
perform frame-skipping, or altering a frequency of read- 
ing the signal captured out of the image pickup device 
and controlling the signal processor to add the image 
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data on a frame basis. 

[001 2] Alternatively, according to the invention, a sub- 
frame number is added to each of the frames of the set 
frame rate included within one frame period of reference 
frame rate. The sub-frame number is included in said 
associated information. This allows the frame-skipping 
to be easily performed when the reproduction speed is 
changed by performing the frame-skipping on frame ba- 
sis. 

[001 3] Further, according to the present invention, the 
signal processor samples analog audio signal to gener- 
ate audio data and the controller controls a sampling fre- 
quency of the analog audio signal in the signal proces- 
sor based on the set frame rate. Then, the transmitter 
combines the associated information with the image da- 
ta and the audio data to transmit the combined ones. 
This allows the audio to be reproduced at a variable 
speed fitting the image. 

[0014] Additionally, the imaging apparatus further 
comprises a signal recording apparatus wherein the 
transmitter transmits a signal combining the associated 
information with the image data to the signal recording 
apparatus recording the signal thus combined on re- 
cording medium. 

[0015] Ttie concluding portion of this specification 
particularly points out and directly claims the subject 
matter of the present invention. However those skill in 
the art will best understand both the organization and 
method of operation of the invention, together with fur- 
ther advantages and objects thereof, by reading the re- 
maining portions of the specification in view of the ac- 
companying drawing(s) wherein like reference charac- 
ters refer to like elements. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0016] 

Fig. 1 is a diagram for illustrating a configuration of 
contents-providing system: 

Fig. 2 is a block diagram showing a configuration of 
imaging apparatus; 

Fig. 3 is a flowchart showing an offer operation of 
sub-frame number; 

Fig. 4 is an illustration illustrating SDI format; 
Fig. 5 is an illustration illustrating SOTI format; 
Fig. 6 is an illustration illustrating SDTI-CP format; 
Fig. 7 is a block diagram showing a configuration of 
another imaging apparatus; 
Figs. 8A through 8E are diagrams showing relation- 
ship (part 1) between the image data and the asso- 
ciated information; 

Figs. 9A through 9E are diagrams showing relation- 
ship (part 2) between the image data and the asso- 
ciated information; 

Fig. 10 is a block diagram showing a configuration 

of signal-recording apparatus; 

Figs. 11 A through 11 C are illustrations each show- 
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ing GUI representation at an editing operation; and 
Figs. 12A through 12J are illustrations showing a 
reproduction operation of the image. 

5 DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 

[0017] Referring to the drawings, the invention will 
now be described in detail with reference to an embod- 

io iment according to the invention. Fig. 1 shows an entire 
configuration of contents-providing system for providing 
the contents such as the contents for audio and/or video. 
An imaging apparatus 10 generates Image data having 
its frame rate variably set and combines associated in- 

is formation Including frame rate information on the frame 
rate of the image data with this image data to supply a 
signal recording apparatus 20 or an editing apparatus 
30 with the combined ones as material data DTm. When 
the imaging apparatus 1 0 is provided with an audio input 

20 apparatus 1 9, it generates audio data to supply the sig- 
nal recording apparatus 20 or the editing apparatus 30 
with the audio data and the associated information as 
material data DTm. 

[0018] The signal recording apparatus 20 receives 
25 the material data DTm and records it on recording me- 
dium. It also reproduces the material data DTm record- 
ed on the recording medium and supplies the editing ap- 
paratus 30 with the reproduced data. The imaging ap- 
paratus 1 0 and the signal recording apparatus 20 may 
30 be composed as a combination camera/recorder sys- 
tem, be combined together each other as so-called 
"dockable" typed camera/recorder system, and be sep- 
arately provided with them being connected through a 
cable. 

35 [0019] The editing apparatus 30 receives the material 
data DTm from the signal recording apparatus 20 or the 
imaging apparatus 1 0 and performs editing process on 
the material data DTm to produce image data and audio 
data the editor wishes to create. The editing apparatus 

*o 30 combines associated information including frame 
rate information with the image data and the audio data 
thus produced to generate contents-data DC for broad- 
casting, distribution, and the like. 
[0020] Fig. 2 is a block diagram showing a confrgura- 

45 tion of the imaging apparatus 10. imaging unit 12 re- 
ceives incident light through imaging lens system 11 and 
an image pickup device such as charge coupled device 
(CCD) mounted within the imaging unit 12 captures sub- 
ject image on its image surface. The image pickup de- 

50 vice generates imaged charge of the subject image 
though photoelectric transfer characteristics thereof. 
The imaging unit 12 also reads out the imaged charge 
thus generated on the basis of driving signal CR from a 
timing generator 142, which will be described later, and 

55 generates imaged signal Sp having a frame rate that 
corresponds to the driving signal CR to supply camera 
processing circuit 131 in signal processor 13 with the 
imaged signal Sp. 
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[0021] Based on timing signal CT received from the 
timing generator 1 42, the camera processing circuit 131 
performs various signal processing at timings synchro- 
nized with the imaged signal Sp. Illustratively, such the 
signal processing represents noise-reduction process- 
ing for reducing noise components from the imaged sig- 
nal Sp by means of correlated dual sampling or the like, 
transformation processing from the imaged signal Sp 
thus noise-reduced to digital image data, clamp 
processing of the image data, processing on shading 
correction and piliing-up of deficiency in the image pick- 
up device, v correction, contour enhancement process- 
ing, knee correction, and the like. Further, the camera 
processing circuit 131 performs other various signal 
processing under the process conditions based on the 
operation control signal CS received from Imaging con- 
trol circuit 141 of controller 14. Thus, the camera 
processing circuit 131 generates image data DV thus 
obtained according to various signal processing and 
supplies transmitting unit 15 with the image data DV. 
[0022] The timing generator 1 42 of controller 1 4 gen- 
erates driving signal CR corresponding to operation 
control signal CS from the imaging control circuit 141 
and supplies the imaging unit 12 with the driving signal 
CR, thereby allowing a reading-out frequency of imaged 
charge in the imaging unit 1 2 to be altered. It, then, also 
controls a frame rate of the imaged signaf Sp to be set 
to a set frame rate FRs based on an operation signal 
PSafrom user interface unit 1 6. For example, frame fre- 
quency, 59.94Hz or 29.97Hz in the case of NTSC sys- 
tem or frame frequency, 50 Hz or 25Hz in the case of 
PAL system is set to the frame frequency of reference 
frame rate FRr and an operation such that the set frame 
rate FRs is set to the one k times the reference frame 
rate FRr is carried out (k is positive value, not limited to 
an integer), if so, a frame rare of the imaged signal Sp 
is controlled to be equal to the one k times the reference 
frame rate FRr. 

[0023] Alternatively, altering a period of reading-out 
pulse (sensor gate pulse) for transferring imaged charge 
accumulated on each pixel from the respective pixels in 
an image pickup device such as CCD to transfer unit or 
the like allows reading-out frequency of imaged charge 
to be altered, thereby altering the frame rate. 
[0024] It may be adapted for Common Data Rate 
(CDR) system in this case. When the CDR system is 
used, available frame rate may be altered, and a frame 
rate of the signal output from the CCD may be constant 
thus making processed rate in the camera processing 
circuit 131 or the like constant. 

[0025] This CDR system has been generally dis- 
closed in PCT Application No. PCT/JP03/0051 filed on 
January 22, 2003. 

[0026] The timing generator 142 also generates tim- 
ing signal CT synchronized with the driving signal CR 
and supplies the.camera processing circuit 131 and au- 
dio signal processing circuit 132 with the timing signal 
CT. The timing generator 142 further generates frame 



rate information DM -FRs indicating the set frame rate 
FRs that is frame rate of image data DV and supplies 
the transmitting unit 15 with the frame rate information 
DM- FRs. The timing generator 142 also generates sub- 
s frame number BN. This sub-frame number BN is iden- 
tification number for identifying each frame included in 
each of the frame periods of the reference frame rate 
FRr when the set frame rate FRs is set to be higher than 
the reference frame rate FRr. The timing generator 142 
io supplies the transmitting unit 15 with this sub-frame 
number BN as frame identification information DM-BN. 
[0027] Fig. 3 is a flowchart showing an offer operation 
of sub-frame number in the timing generator 142. The 
timing generator 142 performs a frequency division on 
w oscillated signal of a predetermined frequency, for ex- 
ample, to set the set frame rate FRs so that the frame 
periods of the reference frame rate FRr and the set 
frame rate FRs may be synchronized with each other, 
thereby generating a frame reference timing indicating 
20 a generation of the driving signal CR and a breakpoint 
between the periods of frames of reference frame rate 
FRr based on the frame period of the set frame rate FRs. 
[0028] At step ST1 , the timing generator 142 deter- 
mines whether or not the frame reference timing is de- 
25 tected. If the frame reference timing is detected, the 
process goes to step ST2. If no frame reference timing 
is detected, the process goes back to the step ST1 . 
[0029] When the frame reference timing is detected 
at the step ST1 , the process goes to the step ST2, as 
so described above, where the timing generator 1 42 initial- 
izes the sub-frame number BN, thus setting the sub- 
frame number BN to an initial value, for example, 0. The 
process then goes to step ST3. 

[0030] At the step ST3, the timing generator 142 de- 
35 term ines whether or not the frame reference timing has 
been detected during a period of time starting from de- 
tecting the frame reference timing and ending before 
one frame of period of the set frame rate FRs has been 
passed. If no frame reference timing is detected, the 
40 process goes to step ST4 where the timing generator 
142 adds 1 to the sub-frame number BN and then up- 
dates the added sub-frame number BN. The process 
then goes back to step ST3. Thus, when no frame ref- 
erence timing has been detected before one frame of 
« period of the set frame rate FRs has been passed, the 
sub-frame number BN is allocated in sequence to the 
set frame rate FRs for each of the frames of periods 
thereof. 

[0031] Thereafter, when the timing generator 1 42 de- 
50 tects the frame reference timing before one frame of pe- 
riod of the set frame rate FRs has been passed, the 
process goes back to the step ST2 where the sub-frame 
number BN is initialized. 

[0032] Every frame of period of the reference frame 
55 rate FRr, the sub-frame number BN may be added to 
the frame image of the set frame rate FRs provided dur- 
ing a frame period of time. 

[0033] Referring back to Fig. 2, the imaging control 
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circuit 141 of the controller 14 is connected with the user 
interface unit 16. When, in the imaging apparatus 10, 
switching operation and alteration operation of the 
frame rate are performed, the user interface unit 1 6 gen- 
erates operation signal PSa corresponding to these op- 
erations and supplies the imaging control circuit 141 
with the operation control signal PS. When receiving the 
operation signal PSa from external equipment such as 
a remote controller, not shown, the user interface unit 
16 also supplies the imaging control circuit 141 with the 
operation signal PSa. 

[0034] The imaging control circuit 141 generates op- 
eration control signal CS to allow the imaging apparatus 
1 0 to be operated according to the operation signal PSa 
based on the operation signal PSa received from the 
user interface unit 16. It also supplies the camera 
processing circuit 131 and the timing generator 142 with 
the operation control signal CS. 

[0035] The audio signal processing circuit 132 re- 
ceives analog audio signal Sin. The audio signal 
processing circuit 132 also performs sampling process 
on the analog audio signal Sin based on timing signal 
CT received from the timing generator 142 to generate 
digital audio data DA and supply the transmitting unit 1 5 
with the digital audio data DA. 

[0036] The transmitting unit 1 5 generates associated 
information DM including the frame rate information 
DM-FRs and the frame identification information DM-BN 
and combines the image data DV and the audio data 
DA with the associated information DM to generate ma- 
terial data DTm. The transmitting unit 15 supplies the 
signal recording apparatus 20 or the editing apparatus 
30 with the materia) data DTm. The associated data DM 
may Include not only information on the set frame rate 
FRs and the sub-frame number BN but also information 
on imaged date and time, imaged condition, imaged 
contents, and the like. 

[0037] It is conceivable that as a way to combine the 
image data DV and the audio data DA with the associ- 
ated information, the associated information DM is in- 
serted into a data stream of image or a header of the 
data stream when the image data DV and the audio data 
DA are compressed to generate the material data DTm 
as the data stream. 

[0038] For example, when a moving picture compres- 
sion system in Moving Picture Experts Group (MPEG) 
standardized as ISO (international Organization for 
Standardization)/! EC (International Eiectrotechnical 
Commission) 13818-2 is utilized, the associated infor- 
mation DM may be inserted into a region for extension 
or the like provided in a picture layer, thereby allowing 
the associated information DM to be inserted on a pic- 
ture basis. Alternatively, when a synchronization multi- 
plexed system in MPEG standardized as ISO/IEC 
1381 8-2 is utilized, the associated information DM may 
be inserted into an optional field or the like provided in 
a header in Packetized Elementary Stream (PES). 
[0039] Further, it is conceivable that when SDI format 



standardized as SMPTE (Society of Motion Picture and 
Television Engineers) 259M, Television- 10-Bit 4:2.2 
Component and 4fsc Composite Digital Signals-Serial 
Digital interface, in order to transfer non -compressed 

5 image and audio data, SDTI format standardized as 
SMPTE 305M, Television -Serial Data Transport Inter- 
face (SDTI), in order to transfer compressed image and 
audio data, or SDTI-CP format standardized as SMPTE 
326M, Television-SDTl Content Package Format (SD- 

10 TI-CP) that obtained by further defining SDTI format is 
utilized, the associated information DM may be inserted 
into a signal having each format as UMID data stand- 
ardized as SMPTE 330M, Television-Unique Material 
Identifier (UM ID). In the present invention, a way to com- 

15 bine the Image data DV and the audio data DA with the 
associated Information is not limited to the above ones. 
Various ways to do so are conceivable. 
[0040] UMID is an identifier for uniquely identifying 
material data such as image data and audio data. UMID 

so defines Basic UMID or Extended UMID that is identifier 
as being obtained by adding Signature Metadata to Ba- 
sic UMID. Basic UMID includes a label for identifying 
digital data, information for indicating whether or not 
overwrite or editing is performed on the material data, a 

25 number for distinguishing the material data from each 
other, and the like. Signature Metadata includes infor- 
mation on material-data-forming date and time, correc- 
tion information (information on difference in time) con- 
cerning material-data-forming time, location information 

30 indicative of latitudes, longitudinal, and altitude, infor- 
mation on name of organization, and the like. Signature 
Metadata also includes a region for user code, to which 
the associated information DM may be stored, for ex- 
ample. 

35 [0041] The Extended UMID thus obtained is inserted 
into each of the signals having various formats. When 
SDI format is utilized, the Extended UMID is inserted 
into an ancillary data region, as shown in Fig. 4. When 
SDTI format is utilized, the Extended UMID is inserted 

40 into an ancillary data region except for a header data 
portion, as shown in Fig. 5. When SDTI-CP format is 
utilized, the data to be inserted is inserted into a payload 
region on an item basis. Specifically, as shown in Fig. 
6, system item composed of information on image and 

45 audio, picture item composed of image data, audio item 
composed of audio data, and auxiliary (AUX) item com- 
posed of other data are inserted in sequence. In the sys- 
tem item, regions for metadata sets such as Package 
Metadata set, Picture Metadata set : audio Metadata set. 

50 and Auxiliary Metadata set are provided to allow the 
metadata to be inserted thereinto. This permits the ex- 
tended UMID to be inserted into these regions. 
[0042] The above imaging apparatus 1 0 alters read- 
out frequency of imaged charge in the imaging unit 12 

55 to generate the material data DTm of a predetermined 
set frame rate FRs, thereby allowing the set frame rate 
FRs to be successively altered. If, however, the set 
frame rate FRs is altered solely in a stepwise, frame- 
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skipping allows the material data DTm of a predeter- 
mined set frame rate FRs to be generated. This Is, gen- 
erating image data DVa having a constant frame rate 
higher than the set frame rate FRs and extracting image 
data of only the set frame rate FRs from the image data 
DVa allows the material data DTm of a predetermined 
set frame rate FRs to be generated. 
[0043] Fig. 7 shows a configuration of another imag- 
ing apparatus 10a performing such a generation. In Fig. 
7, like reference numbers refer to like elements shown 
in Fig. 2 and thus, the detailed description thereof is 
omitted. 

[0044] Timing generator 182 in controller 18 gener- 
ates driving signal CRa corresponding to the highest 
value of the set frame rate FRs that is set through user 
interface unit 16 and supplies the imaging unit 12 with 
the driving signal CRa. Based on the driving signal CRa, 
the imaging unit 12 generates imaged signal Spa having 
a fixed frame rate FRq higher than a reference frame 
rate. The imaging unit 1 2 then supplies camera process- 
ing circuit 131 with the Imaged signal Spa. When the set 
frame rate FRs may be altered up to n times (where n 
is positive number) the reference frame rate FRr, the Im- 
aging unit 12 generates the Imaged signal Spa having 
a frame rate n times the reference frame rate FRr, and 
supplies the camera processing circuit 131 with the im- 
aged signal Spa. In other words, the imaging unit 12 
generates the imaged signal Spa having a fixed frame 
rate without any influence of the set frame rate FRs that 
Is set through the user interface unit 16. 
[0045] The timing generator 1 82 also generates tim- 
ing signal CTa synchronized with the driving signal CRa 
and supplies the camera processing circuit 131 and au- 
dio signal processing circuit 132 in signal processor 17 
and effective frame signal generation circuit 183 in the 
controller 18 with the timing signal CTa. 
[0046] The camera processing circuit 131 generates 
image data DVa of fixed frame rate FRq based on the 
imaged signal Spa and supplies the effective data se- 
lection circuit 171 with the image data DVa. The audio 
signal processing circuit 132 carries out a sampling 
based on the timing signal CTa of a fixed frequency to 
generate audio data DAa and supplies the effective data 
selection circuit 171 with the audio data DAa. 
[0047] Imaging control circuit 181 generates a set in- 
formation signal CF indicating a set frame rate FRs 
based on operational signal PSa received from the user 
interface unit 1 6 and supplies the effective frame signal 
generation circuit 1 83 with the set information signal CF. 
[0046] The effective frame signal generation circuit 
1 83 extracts data on a frame basis from the image data 
DVa based on a ratio of the fixed value of frame rate 
FRq of the image data DVa to a value of the set frame 
rate FRs indicated in the set information signal CF and 
generates extraction control signal CC for generating 
image data DV of the set frame rate FRs. The effective 
frame signal generation circuit 183 also synchronizes 
the extraction signal CC with the timing signal CTa and 



supplies the effective data selection circuit 171 with the 
synchronized extraction control signal CC. When the 
frame rate FRq of the image data DVa is n times the 
reference frame rate FRr and the set frame rate FRs is 
s n/2 times the reference frame rate FRr, the effective 
frame signal generation circuit 1 83 generates the ex- 
traction control signal CC for controlling data extraction 
on a frame basis every other frame from the image data 
DVa and supplies the effective data selection circuit 1 71 
10 with the extraction control signal CC synchronized with 
the timing signal CTa. 

[0049] The effective frame signal generation circuit 
183 further generates frame rate information DM-FRs 
indicating the set frame rate FRs based on the set infor- 
ms mation signal CF and supplies the transmitting unit 15 
with the frame rate Information DM-FRs. Since the ef- 
fective frame signal generation circuit 183 may deter- 
mine a number of frame during a period of frame with 
the reference frame rate FRr according to the extraction 
20 control signal CC, the effective frame signal generation 
circuit 183 sets sub-frame number on a frame during a 
period of each frame with the reference frame rate FRr. 
The effective frame signal generation circuit 183 then 
supplies the transmitting unit 15 with this sub-frame 
25 number BN as the frame identification information 
DM-BN. 

[0050] The effective data selection circuit 1 71 extracts 
the image data DVa and the audio data DAa of the frame 
indicated by the extraction control signal CC and sup- 
30 pliesthetransmittingunit15withthemastheimagedata 
DV and the audio data DA. It is conceivable that the ef- 
fective frame signal generation circuit 183 may supply 
the effective data selection circuit 171 with the frame 
rate information DM-FRs indicating the set frame rate 
35 FRs, and the effective data selection circuit 171 may 
perform frame-skipping on the audio data DAa accord- 
ing to a ratio of the set frame rate FRs to a frame rate 
when the audio data DAa is generated. When the frame 
rate FRq in generating the audio data DAa is n times the 
40 reference frame rate FRr and the set frame rate FRs is 
n/2 times the ref eren ce frame rate FRr, the effective data 
selection circuit 1 71 performs the frame-skipping on the 
audio data DAa every other sample. In this case, since 
an interval of the frame-skipping may be shortened as 
45 compared by a case where performing the frame-skip- 
ping on the audio data DAa on a frame basis, audio hav- 
ing an excellent sound quality may be get based on the 
audio data DA. 

[0051] Thus, the image data DVa having a fixed frame 
50 frequency makes unnecessary the alteration of opera- 
tion frequencies in the imaging unit 12 and the camera 
processing circuit 131 of the signal processor 17. This 
allows configurations of the imaging unit 12 and the 
camera processing circuit 131 to be made simpler. 
55 Since only the data extraction on a frame basis from the 
image data DVa may generate image data DV of the set 
frame rate FRs, generating the image data DV of a pre- 
determined set frame rate FRs from the image data Dva 
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is easily allowed. 

[0052] When the imaging apparatus is provided with 
video memory or an adder and a divider, it may generate 
the image data DV by adding the image data every the 
predetermined frames. This allows a variable range of s 
frame rate in the imaged signal Sp to be limited and the 
frame rate FRs to be successively altered. Adding the 
Imaged signal Sp of n frames and dividing the signal lev- 
el by n allows the signal having a frame rate 1/n times 
the imaged signal Sp to be obtained even if a frame rate 10 
of the imaged signal Sp is not divided by n. Further, suc- 
cessively altering the read-out frequency of imaged 
charge allows the set frame rate FRs to be successively 
altered. 

[0053] Figs. 8A through 8E and 9A through 9E are di- « 
agrams showing relationship between the image data 
DV generated by the imaging apparatus 10, 10a and the 
associated information DM. 

[0054] When the set frame rate FRs is set to the one 
equal to or twice the reference frame rate FRr as shown 
in Fig. 8A, the associated information DM that includes 
the frame rate Information DM -FRs indicating the set 
frame rate FRs, as shown in Fig. 8C, and the frame iden- 
tification information DM-BN indicating the sub-frame 
number BN, as shown in Fig. 8D is combined wfth image 
data DV as shown in Fig. 8B (in Fig. 8B, frame images 
based on the image data DV are shown). Fig. 8E illus- 
trates a relationship between time and the frame imag- 
es. The frame rate information DM-FRs may include a 
magnification of the set frame rate FRs to the reference 
frame rate FRr in addition to the set frame rate FRs. The 
frame rate information DM-FRs is indicated by the mag- 
nification in Figs. BE, 9C, and 12B. 
[0055] When the set frame rate FRs is set to the one 
equal to or half as much as the reference frame rate FRr 
as shown in Fig. 9A, the associated information DM that 
includes the frame rate information DM-FRs indicating 
the set frame rate FRs, as shown in Fig. 9C, and the 
frame identification information DM-BN indicating the 
sub-frame number BN, as shown in Fig. 9D is combined 
with image data DV as shown in Fig. 9B (in Fig. 9B> 
frame images based on the image data DV are shown). 
Fig. 9E illustrates a relationship between time and the 
frame images. 

[0056] Fig . 1 0 shows a co nf igu ration of s ign al- record- 
ing apparatus 20 such as a videotape recorder. 
[0057] When the material data DTm that combines 
main data indicating image and/or audio with the asso- 
ciated information including frame rate information of 
the main data is input, encoder 211 of signal-recording 50 
processor 21 receives the material data DTm as record- 
ing data. The encoder 211 produces error-correcting 
codes and performs data-shuffling, multiplexing, chan- 
nel coding, and the like based on operation control sig- 
nal CTM received from a recording/reproducing control 55 
unit 24 using the material data DTm thus received so 
that the encoder 211 can generate recording signal SW. 
The encoder 211 then supplies a terminal 212a of a 



40 



45 



changeover switch 212 with the generated recording 
signal SW. 

[0058] The recording/reproducing control unit 24 sup- 
plies the changeover switch 21 2 with a switch control 
signaJ CTW. Operations of the changeover switch 212 
are controlled based on the switch control signal CTW 
and thus, the recording signal SW is supplied to record- 
ing amplifier 213a or 213b through terminal 212b or 
212c of the changeover switch 212. 
[0059] The recording amplifier 21 3a receives and am- 
plifies the recording signal SW and supplies signai- 
switching device 221a of signal recording/reproducing 
unit 22 with the amplified signal SW. The recording am- 
plifier 213b receives and amplifies the recording signal 
SW and supplies signal-switching device 221b of the 
signal recording/reproducing unit 22 with the amplified 
signal SW. 

[0060] When recording a signal, the signal-switching 
device 221 a supplies magnetic head 222a with the am- 
plified recording signal SW received from the recording 
amplifier 213a based on a changeover control signal 
CWR received from the recording/reproducing control 
unit 24. When reproducing a signal, the signal obtained 
by the magnetic head 222a Is supplied to reproducing 
amplifier 231a of the signal reproduction processor 23. 
Similarly, when recording a signal, the signal-switching 
device 221 b supplies magnetic head 222b with the am- 
plified recording signal SW received from the recording 
amplifiers 213b based on a changeover control sfgnai 
CWR received from the recording/reproducing control 
unit 24. When reproducing a signal, the signal obtained 
by the magnetic head 222b is supplied to reproducing 
amplifier 231 b of the signal reproduction processor 23. 
[0061] Thus, supplying the magnetic heads 222a, 
222b built in a rotating head drum (not shown) wfth the 
amplified recording signal SW allows image data and 
audio data, and the associated information to be record- 
ed on magnetic tape, which is not shown. The magnetic 
heads 222a, 222b read out the signal recorded on the 
magnetic tape and supplies the reproducing amplifiers 
231a. 231b, respectively, with the read-out signal. 
[0062] The reproducing amplifier 231a receives and 
amplifies a signal obtained by the magnetic head 222a 
and supplies a terminal 232a of a changeover switch 
232 with the amplified signal. The reproducing amplifier 
231b receives and amplifies a signal obtained by the 
magnetic head 222b and supplies a terminal 232b of the 
changeover switch 232 with the amplified signal. A ter- 
minal 232c of the changeover switch 232 is connected 
with a decoder 233. 

[0063] The recording/reproducing control unit 24 sup- 
plies the changeover switch 232 with a switch control 
signal CTR. Operations of the changeover switch 232 
are controlled based on the switch control signal CTR 
on which the signals output from the recording amplifiers 
213a, 213b, respectively, are selected. The selected 
signal is supplied to the decoder 233 as reproduced sig- 
nal SR. The decoder 233 decodes the reproduced sig- 
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nal SR and performs data separation, de-shuffling, an 
error-correcting processing, and the like based on op- 
eration control signal CTM received from the recording/ 
reproducing control unit 24. Thus, the decoder 233 gen- 
erates the material data DTm that combines main data 
indicating image and/or audio with the associated infor- 
mation including frame rate information of the main data 
and then transmits the combined ones. 
[0064] The recording/reproducing control unit 24 is 
connected with interface unit 25. The recording/repro- 
ducing control unit 24 receives operational signal PSv 
from the interface unit 25 or another operational signal 
PSw from an external apparatus such as the editing ap- 
paratus 30. The recording/reproducing control unit 24 
also generates the switch control signals CTW, CTR and 
the operation control signals CTM, CTM controlling op- 
erations of the encoder 211 and the decoder 233, re- 
spectively, based on the operational signal PSv or PSw. 
The recording/reproducing control unit 24 further per- 
forms a driving control of a magnetic tape or a rotation 
head. The recording/reproducing control unit 24 gener- 
ates a display signal PH and supplies display unit 26 
with It to display an operational condition of a video tape 
recorder, and various Information and the like. 
[0065] Although a case where the material data DTm 
is recorded on the magnetic tape has been shown in 
Fig. 10, the material data may be recorded on disk-like 
optical or magnetic recording medium or signal-repro- 
duction device using semiconductor storage element. 
[0066] The signal-recording apparatus 20 is enough 
if the image data DV and audio data DA are combined 
with the associated information DM and they are record- 
ed or transmitted. Thus, the material data DTm to be 
input is not limited to the one having the associated In- 
formation inserted into the data stream, header, or the 
like. The material data may be supplied through different 
signal lines. 

[0067] Combining the main data with the associated 
information DM allows editing processing and variable 
speed reproduction to be easily performed using the da- 
ta generated by the imaging apparatus 10, 10a or the 
data recorded on the recording medium by the signal- 
recording apparatus 20. 

[0068] During an editing process of the data combin- 
ing the main data with the associated information DM. 
a speed available for a reproduction may be set based 
on the set frame rate FRs. This speed is set so that a 
natural reproduction image can be realized through a 
frame-skipping of a predetermined frame period or a 
frame repetition. If a multiple of the set frame rate FRs 
to the reference frame rate FRris more than one, name- 
ly, FRs/FRr>1, the speed available for reproduction is 
set by searching for the divisors of this multiple other 
than one and calculating the reciprocals of the searched 
divisors. If the set frame rate FRs is 10 times the refer- 
ence frame rate FRr t the divisors of this multiple, 1 0 oth- 
er than one are of 2 S 5, and 1 0 and thus, the reciprocals 
thereof are of 1/2, 1/5, and 1/10, respectively. Selection 



of one of the reciprocals of divisors allows the speed 
available for reproduction to be set to less than the same 
speed as normal reproduction speed. When the speed 
available for reproduction not less than the same speed 
5 as the normal reproduction speed is set to positive inte- 
ger multiples of the normal reproduction speed, the 
frame-skipping periods in each of the speeds available 
for reproduction may be equal to each other. 
[0069] If a multiple of the set frame rate FRs to the 
io reference frame rate FRr is less than one, namely, FRs/ 
FRr<1 , the speed available for reproduction is set by 
searching for the reciprocals of this multiple and calcu- 
lating the divisors of the searched reciprocals other than 
one or integer multiples of the searched reciprocals. If 
the set frame rate FRs Is 1/6 times the reference frame 
rate FRr, the divisors of the searched reciprocals other 
than one are of 2, 3, and 6 and the Integer multiples of 
the reciprocals are of 6, 12, 18, — , respectively. Selec- 
tion of one of them allows the speed available for repro- 
20 duction to be set. This allows frame-skipping periods or 
numbers of the frame repetitions in each of the speeds 
available for reproduction may be equal to each other. 
The speed available for reproduction of less than the 
same speed as the normal reproduction speed Is set to 
25 the one/positive multiples the normal reproduction 
speed. This allows numbers of the frame repetitions In 
each of the speeds available for reproduction to be 
equal to each other. Since, however, this is concerned 
with a repetition of images, a lower limit of the speed 
30 available for reproduction may be set to the same speed 
as the normal reproduction speed. 
[0070] Figs. 1 1 A through 11 C illustrate GUI represen- 
tations each for an editing operation using the associat- 
ed information. For example, in each of the GUI repre- 
ss sentations, a material administration browser window 
401 as material administration display is provided at an 
upper-left side thereof; a story board window 402 as re- 
production order display is provided at an lower- left side 
thereof; a monitor viewer window 403 as reproduced im- 
<to age display for displaying an image before or after the 
editing process is provided at an upper-right side there- 
of; and a time line window 404 as reproduction time or- 
der display is provided at a lower-left side thereof. An 
operation control bar 405 is provided between the mon- 
45 itor viewer window 403 and the time line window 404. 
[0071] Fig. 11 A illustrates the GUI representation 
where the reproduction speed is the same speed as the 
normal reproduction speed. Fig. 1 1 B illustrates the GUI 
representation where the reproduction speed is a tenth 
50 the normal reproduction speed. Fig. 11C illustrates the 
GUI representation where the reproduction speed is 
twice as much as the normal reproduction speed. Loca- 
tions and sharps of the material administration browser 
window 401 , the story board window 402, the monitor 
55 viewer window 403, the time line window 404, and the 
operation control bar 405 are respectively illustrative 
ones and thus, of course, they are not limited to the 
above locations and sharps. 
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[0072] The material administration browser window 
401 shows a list of Items of the material data available 
for editing. For each item of the material data, a stamp 
view (a thumbnail view) for indicating a title, a length, 
contents of items of the stored material data is shown. 
[0073] The story board window 402 serves as a work- 
ing area for a production of contents. Arranging the 
items of material data on the reproduced order allows 
the contents to be produced. 

[0074] The monitor viewer window 403 represents not 
only an image based on the material data but also a lo- 
cation of reproduction and a variable speed bar for indi- 
cating a variable range of reproduction speed. 
[0075] The time line window 404 serves as a working 
area for a production of contents according to a more 
detailed method to allocate items of the material data 
along a time axis. 

[0076] The operation control bar 405 represents an 
operation key for reproducing the items of material data 
and the contents arranged on the time line window 404. 
[0077] When reproducing the selected material data, 
the set frame rate FRs is determined on the basis of the 
associated information DM, and then, such a calculation 
that the reproduction speed FP multiplied by the set 
frame rate FRs equals a determined value FO is per- 
formed. Reproduction process conditions are deter- 
mined on the determined value FD. When the reproduc- 
tion speed is the same speed as the normal reproduc- 
tion speed as shown in Fig. 11 A and the set frame rate 
FRs is 1 0 times the reference frame rate FRr, the repro- 
duction speed FP multiplied by the set frame rate FRs 
equals the determined value FD as (1*10=10). 
[0078] Fig. 1 2A illustrates images based on the image 
data DV where the set frame rate FRs is 10 times the 
reference frame rate FRr. Fig. 12B illustrates the frame 
rate information DM -FRs for indicating the set frame rate 
FRs of each of the frame images. Fig. 12C illustrates 
theframe identification information DM-BN for indicating 
the sub-frame numbers. Fig. 12D illustrates the absolute 
frame numbers AN of the frame images. Since the set 
frame rate FRs is 10 times the reference frame rate FRr, 
there are 10 frame images of the set frame rate FRs 
within one frame period of the reference frame rate FRr, 
thereby repeating the sub-frame numbers of zero 
through nine. 

[0079] When the determined value FD is set to 10, 
namely, FD=1 0, as shown in Figs. 12E through 1 2G, the 
image signal Svm is generated using every 10 frames, 
namely, with the image data of nine frames being 
skipped. This allows reproduced image having same 
speed as the normal reproduction speed to be repre- 
sented on the monitor viewer 403 based on the image 
signal Svm. Fig. 12E illustrates the frame identification 
information DM-BN for indicating the sub-frame num- 
bers. Fig. 12F illustrates the absolute frame numbers 
AN of the frame images. Fig. 12G illustrates frame im- 
ages represented by the image signal Svm. 
[0080] An operation is carried out such that an indi- 



cated width of a frame image representation region 
404a in the time line window 404 of Fig. 1 1 A is enlarged 
along a direction indicated by an arrow A or a sliding 
cursor 404d represented by a short bold line in a console 

s indication 403c for variable speeds in the monitor viewer 
403 is slid on a direction indicated by an arrow B. in this 
case, a processing as the slow motion of reproduction 
is performed. For example, when the speed available 
for reproduction is set to 1/10 times the normal repro- 

10 duction speed, a reproduction speed indication 403b in 
the monitor viewer 403 is altered to "x 0.1" as shown in 
Fig. 11B. At the same time, the sliding cursor 404d is 
slid to a cursor position representing a tenth times the 
normal reproduction speed In the console indication 

is 403c. Since the long reproduction time is required, the 
Indicated width of the frame image representation re- 
gion 404a Is also enlarged, as shown in Fig. 11 B. 
[0081] If the speed available for reproduction is set to 
1/10 times the normal reproduction speed as shown in 

20 Fig. 1 1 B, the determined value FD is set to one, namely, 
FD=10*(1/10)=1 . When the determined value FD is set 
to one, namely, FD=1, the image signal Svm Is gener- 
ated using every FD=1 frame of the image data DV, 
namely, without any frame being skipped. This allows 

25 the reproduced images having a tenth times the normal 
reproduction speed to be represented on the monitor 
viewer 403 based on the image signal Svm, as shown 
in Fig. 12A. 

[0082] An operation is carried out such that an indi- 
30 cated width of the frame image representation region 
404a in the time line window 404 of Fig. 1 1 A is shortened 
along a direction indicated by an arrow B or a sliding 
cursor 404 d in the console indication 403c is slid on a 
direction indicated by an arrow A in the monitor viewer 
35 403. In this case, a processing as the quick motion of 
reproduction is performed. For example, when the 
speed available for reproduction is set to twice as much 
as the normal reproduction speed, a reproduction speed 
indication 403b in the monitor viewer 403 is altered to 
40 -x 2.0" as shown in Fig. 11 C. At the same time, the slid- 
ing cursor 404d is slid to a cursor position representing 
twice of the normal reproduction speed in the console 
indication 403c. Since the short reproduction time is re- 
quired, the indicated width of the frame image represen- 
ts tation region 404a is shortened, as shown in Fig. 11 C. 
[0083] If the speed available for reproduction is set to 
twice of the normal reproduction speed as shown in Fig. 
11C, the determined value FD is set to 20, namely, 
FD=1 0*2=20. When the determined value FD is set to 
50 20, namely, FD=20, the image signal Svm is generated 
using every 20 frames, namely, with the image data of 
nineteen frames being skipped. This allows the repro- 
duced images having twice speed as much as the nor- 
mal reproduction speed to be represented on the mon- 
55 itor viewer 403 based on the image signal Svm, as 
shown in Figs. 12H through 12J. Fig. 12H illustrates the 
frame identification information DM-BN for indicating the 
sub-frame numbers. Fig. 121 illustrates the absolute 
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frame numbers AN of the frame images. Fig. 12J illus- 
trates frame images represented by the image signal 
Svm. 

[0084] Thus, the material data of the set frame rate 
FRs fs altered to various rates using the associated in- 
formation DM and the altered material data is combined 
with each other, thereby allowing predetermined con- 
tents-data for broadcasting, a distribution, and the like 
to be easily generated. Further, when the edited image 
data is combined with the corresponding associated in- 
formation to generate the contents-data, the user side 
performs the same processing as the one performed in 
the editing apparatus using the associated information 
Included in the contents-data, thereby allowing the re- 
production speed of contents to be altered at the user 
side. For example, the set frame rate FRs is set to the 
one higher than the reference frame rate FRr to gener- 
ate the material data such as sports event casting and 
then, using the material data, the contents-data includ- 
ing the associated information is generated. This allows 
only a predetermined scene to be seen in a slow motion 
by the user with other scenes being seen In normal re- 
production speed. 

[0085] The Invention has been described in detail with 
reference to specific embodiments of the invention in the 
form of the imaging apparatus 10 having a communica- 
tion function. It should be understood that the invention 
is not be limited to the embodiments and that the inven- 
tion can be applied equally well to other types of elec- 
tronic apparatuses. While the foregoing specification 
has described preferred embodiments) of the present 
invention, one skilled in the art may make many modifi- 
cations to the preferred embodiment without departing 
from the invention in its broader aspects. The appended 
claims therefore are intended to cover all such modifi- 
cations as fall within the true scope and spirit of the in- 
vention. 

[0086] Various respective aspects and features of the 
invention are defined in the appended claims. Features 
from the dependent claims may be combined with fea- 
tures of the independent claims as appropriate and not 
merely as explicitly set out in the claims. 



Claims 

1 . An imaging apparatus comprising: 

imaging device for reading a signal captured 
out of an image pickup device so as to generate 
an image signal based on captured image; 
signal processor for generating image data 
based on said image signal; 
controller for controlling operation of at least 
one of said imaging device and said signal 
processor to set a variable frame rate of said 
image data to a set frame rate, and generating 
associated information for indicating at least 
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said set frame rate; and 
transmitter for combining said associated infor- 
mation with said image data to transmit the 
combined ones. 

The imaging apparatus according to Claim 1, 
wherein said controller allows for setting the varia- 
ble frame rate of said image data by means of al- 
tering a frequency of reading the signal captured out 
of said image pickup device. 

The imaging apparatus according to Claim 1, 
wherein said controller allows for setting the varia- 
ble frame rate of said image data by means of con- 
trolling said signal processor to perform frame-skip- 
ping. 

The imaging apparatus according to Claim 1, 
wherein said controller allows for setting the varia- 
ble frame rate of said image data by means of al- 
tering a frequency of reading the signal captu red out 
of said image pickup device and controlling said sig- 
nal processor to add said image data on a frame 
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5. The imaging apparatus according to Claim 1, 
wherein said controller allows for adding a sub- 
frame number to each of the frames of said set 
frame rate included within one frame period of ref- 
erence frame rate so as to include said sub-frame 
number in said associated information. 

6. The imaging apparatus according to Claim 1, 
wherein said signal processor samples analog au- 
dio signal to generate audio data; 

wherein said controller controls a sampling 
frequency of said analog audio signal in said signal 
processor based on said set frame rate; and 

wherein said transmitter combines said asso- 
ciated information with said image data and said au- 
dio data to transmit the combined ones. 

7. The imaging apparatus according to claim 1 . further 
comprising a signal recording apparatus, wherein 
said transmitter transmits a signal combining said 
associated information with said image data to said 
signal recording apparatus recording the signal 
thus combined on recording medium. 

so 8. An imaging method comprising the steps of: 

generating image data based on a signal read 
out of an image pickup device; 
setting a variable frame rate of said image data 
55 to a set frame rate; 

generating associated information including 
frame rate information for indicating said set 
frame rate; and 
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combining said associated information with 
said image data to transmit the combined ones. 

9. The imaging method according to Claim 8, wherein. 

in said step of setting, the variable frame rate of said s 
image data is set by means of altering a frequency 
of reading the signal out of said image pickup de- 
vice. 

1 0. The imaging method according to Claim 8, wherein, 1 o 
in said step of setting, the variable frame rate of said 
image data is set by means of performing frame- 
skipping. 

11. The Imaging method according to Claim 8, wherein. 1* 
In said step of setting, the variable frame rate of said 
image data is set by means of altering a frequency 

of reading the signal out of said image pickup device 
and controlling said signal processor to add said im- 
age data on a frame basis. 20 

12- The imaging method according to Claim 8, wherein 
said associated information includes a sub-frame 
number allocated to each of the frames of said set 
frame rate included within one frame period of ref- 25 
erence frame rate. 

13. The imaging method according to Claim 8, further 
comprising the steps of: 

30 

sampling analog audio signal to generate audio 
data; and 

controlling a sampling frequency of said analog 
audio signal based on said set frame rate, 

35 

wherein, in said combining and transmitting 
step, said associated information is combined with 
said image data and said audio data to transmit the 
combined ones. 

40 

14. The imaging method according to Claim 8, wherein, 
in said combining and transmitting step, a signal 
combining said associated information with said im- 
age data is transmitted to signal recording appara- 
tus recording the signal thus combined on recording *s 
medium. 
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GTblack borders 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 
EDTEXTT5R DEAWmG 

□Blurred OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE^) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: ^^^^ 

IMAGES ARE BEST AVAILABLE COPY 

As rescanning these documents will not correct the image 

fh7^i*^ £ CaSe d ° n0t rep ° rt these * robIems to 
the IFW Image Problem Mailbox. 



